Skip to content

runtime: recoverable hardware faults with fault-site stacks (dynamic libunwind + FP chain)#2028

Open
cpunion wants to merge 4 commits into
xgo-dev:mainfrom
cpunion:codex/fault-unwind
Open

runtime: recoverable hardware faults with fault-site stacks (dynamic libunwind + FP chain)#2028
cpunion wants to merge 4 commits into
xgo-dev:mainfrom
cpunion:codex/fault-unwind

Conversation

@cpunion

@cpunion cpunion commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Based directly on #2019 (contains #2012/#2016): hardware faults become recoverable Go panics, and the unrecovered traceback shows the fault-site chain — C frames down through the Go callers — captured from the signal ucontext with a dynamically-resolved libunwind, falling back to the frame-pointer chain.

Productizes the codex/exp-dyn-unwind experiment (feasibility validated on darwin/arm64 + linux/amd64 + linux/arm64).

What changes

  • SA_SIGINFO fault handler for SIGSEGV/SIGBUS/SIGFPE (x86 integer division previously core-dumped; now recovers with gc's "integer divide by zero"). The interrupted pc/fp comes from the ucontext — the handler's own FP chain dead-ends at the signal trampoline, which is why plain handlers could never see fault stacks. SA_NODEFER because recovery longjmps out through savemask=0 jmpbufs: an auto-blocked signal would stay blocked and turn the next genuine fault into a forced core.
  • Dynamic libunwind tier (_wrap/dynunwind.c): resolved via dlopen/dlsym at install time only — readelf -d stays free of libunwind; LLGO_DYNUNWIND=0 forces the FP walk. Three flavors: darwin libSystem (in-memory name resolution, on by default), linux nongnu (_ULx86_64_*-prefixed; its unw_get_proc_name reads .symtab and can name static C symbols, but does so via /proc/self/maps + ELF mmap — not async-signal-safe — so name capture is opt-in via LLGO_DYNUNWIND_NAMES=1; frame pcs and the resume fp are always captured), linux LLVM (context translated; frames only). Where unwind info runs out, the walk resumes along the FP chain from libunwind's final cursor.
  • Safety: handler work is limited to cursor stepping and register reads; flavor resolution, lazy-state warm-up, and page-size lookup happen at install, and errno is preserved across both installation and the msync readability probe. Fault-context walks probe page readability before dereferencing — an arithmetic-valid frame pointer can still point into an unmapped hole, and faulting inside the fault path would recurse; a re-entered handler restores the default disposition for one clean core. The errno preservation is load-bearing: the nongnu warm-up leaves errno=ENOENT (vdso has no on-disk file), which surfaced as a spurious non-nil err on the first two-result cgo call (v, err := C.f()) and broke _demo/go/cgo on any system with libunwind installed.
  • Traceback: a new PanicTraceback hook prints gc-style frames for fault panics (funcinfo tables for Go frames, libunwind names for dot-less C symbols); non-fault panics keep the existing clite dump.

Decisive result (linux/amd64, C helpers built -fomit-frame-pointer, LLGO_DYNUNWIND_NAMES=1)

FP-only walk: 2 frames, fault pc misattributed to a neighboring Go symbol. Dynamic path — verbatim output of the nofp probe (chain: fx.main -> viaGoNofp -> cexc_nofp_segv -> cexc_nofp_mid x3 -> cexc_nofp_leaf, fault in leaf):

panic: runtime error: invalid memory address or nil pointer dereference

goroutine 1 [running]:
cexc_nofp_mid(...)
	pc=0x293dde
cexc_nofp_mid(...)
	pc=0x293dcb
cexc_nofp_mid(...)
	pc=0x293dcb
cexc_nofp_segv(...)
	pc=0x293df5
fx.main(...)
	/root/fx2/nofp_run.go:14 +0x6f
exit status 2

C frames carry no line-table entry, so the indent line prints the raw pc (offline-resolvable with addr2line/atos) instead of a ???:0 placeholder, and the walk stops after the module's frames — the startup tail (main, __libc_start_main, _start) is plumbing gc tracebacks never show either. A fault inside a named C-library function still prints its leading out-of-text frames.

Reading the frames: cexc_nofp_leaf is inlined into cexc_nofp_mid by the C compiler, so the fault pc reports as mid+0x1e; the next two mid frames are the recursion's return addresses; viaGoNofp is inlined into fx.main by llgo, whose frame carries the exact Go file:line. These are non-exported C symbols — invisible to dladdr — named via nongnu's .symtab reader. With names off (the linux default) the frame count, order and Go frames are identical, but the C pcs fall back to the table's nearest-below attribution and print a confidently wrong Go symbol (verbatim):

github.com/goplus/llgo/runtime/internal/runtime.memequalptr(...)
	.../alg.go:192 +0x2c        <- actually cexc_nofp_mid
(x4, then:)
fx.main(...)
	/root/fx2/nofp_run.go:14 +0x6f

This is the pre-documented C-in-Go-text-range hole; the fix (hole sentinels so foreign pcs print as unknown/named instead of a neighboring Go function) is the P4 follow-up in #2004. Until then, LLGO_DYNUNWIND_NAMES=1 gives correct C names on the nongnu flavor at the cost of async-signal-unsafe work in a dying process's handler.

Known limitation

The handler is installed from the runtime shim's init (lib/runtime); a program that imports nothing which links the public runtime package keeps the pre-existing behavior for C faults (raw core dump — same as before this PR, so no regression, but such programs do not gain fault recovery either). Moving the install into the runtime core is possible if the startup dlopen probes are acceptable for every binary.

Relationship to the other stage-5 PRs

Overlaps with the fault half of #2026 (which also carries Go-panic snapshots and recovered-debug.Stack splicing on the #2023 base); whichever lands second rebases to drop its duplicated fault machinery. Naming normalization (main. prefix) and pcline conformance live in #2023 and apply on top once merged.

Validation

  • macOS: full test/go green; probes verified with the libSystem flavor engaged (LLGO_DYNUNWIND_DEBUG=1).
  • linux/arm64 container: all _demo/go demos, fault regressions, statement suites green.
  • linux/amd64 (qemu): recover + traceback probes, plus the -fomit-frame-pointer comparison above.
  • _demo/go/cgo errno regression counterfactually verified on linux/amd64 with nongnu libunwind installed: fails on the previous head, passes with the errno preservation; also passes with libunwind absent.
  • New regressions: TestCFaultRecoverable (three sequential faults — exercises SA_NODEFER), TestCFaultTraceback (C frame-name assert darwin-only).

🤖 Generated with Claude Code

@codecov

codecov Bot commented Jul 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

cpunion and others added 4 commits July 8, 2026 19:17
…libunwind + FP chain)

Hardware faults — SIGSEGV/SIGBUS and previously-fatal SIGFPE — now
convert to ordinary recoverable Go panics, and the unrecovered traceback
shows the fault-site chain: C frames down through the Go callers.

- A SA_SIGINFO handler captures the interrupted context; the handler's
  own frame-pointer chain dead-ends at the signal trampoline, which is
  why the ucontext pc/fp is required.
- The capture prefers a dynamically-resolved libunwind (dlopen/dlsym at
  install time only — no link-time -lunwind; LLGO_DYNUNWIND=0 disables):
  DWARF/compact-unwind stepping survives C frames compiled without frame
  pointers, and the nongnu flavor's unw_get_proc_name reads .symtab,
  naming static C symbols dladdr cannot see (they otherwise display
  under a neighboring Go function via nearest-below). Where unwind info
  runs out, the walk resumes along the FP chain from libunwind's final
  cursor. Flavors: darwin libSystem, linux nongnu (arch-prefixed
  symbols), linux LLVM (context translated).
- Only man-page async-signal-safe unw_* calls run in the handler
  (resolution and a lazy-state warm-up happen at install). Fault-context
  walks probe page readability (msync) before dereferencing — an
  arithmetic-valid frame pointer can still point into an unmapped hole,
  and faulting inside the fault path would recurse; a re-entered handler
  restores the default disposition for one clean core.
- The unrecovered dump goes through a new PanicTraceback hook (gc-style
  frames via the funcinfo tables; libunwind's name for dot-less C
  symbols); non-fault panics keep the existing clite dump.

Verified: darwin/arm64 (libSystem flavor); linux/amd64 with a
-fomit-frame-pointer C chain — the FP-only walk recovers 2 frames with a
misattributed name, the dynamic path recovers the full chain with
correct static names.

Overlaps with the fault half of xgo-dev#2026 (panic-site snapshots); whichever
lands second rebases to drop its duplicate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Preserve errno across fault-handler installation: the dlopen probes for
  libunwind flavors leave errno=ENOENT on systems without libunwind, and
  the leaked value surfaced as a spurious non-nil err on the first
  two-result cgo call (broke _demo/go/cgo on CI ubuntu). Also preserve
  errno in llgo_mem_readable (msync probe).
- Prime the page size at handler install instead of lazily calling
  sysconf inside the signal handler.
- Capture frame symbol names in the handler only where the resolver is
  in-memory (darwin libSystem). nongnu's unw_get_proc_name reads
  /proc/self/maps and mmaps ELF .symtab — not async-signal-safe — so it
  is now opt-in via LLGO_DYNUNWIND_NAMES=1; frame pcs and the resume fp
  are still captured.
- Move errno/mman/unistd includes and the llgo_pagesz declaration above
  their first use.
C frames have no line table entry; print 'pc=0x<addr>' (offline-resolvable
via addr2line/atos) instead of the '???:0' placeholder. Stop the walk at
the first out-of-text frame once the module's own frames have printed:
everything below main (__libc_start_main, _start) is startup plumbing.
Leading out-of-text frames still print, so a fault inside a named libc/
C-library function keeps its context.
@cpunion cpunion force-pushed the codex/fault-unwind branch from 3351dfa to 2e095ae Compare July 8, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant